Run-Length FM-index

The FM-index is a succinct text index needing only O(Hkn) bits of space, where n is the text size and Hk is the kth order entropy of the text. FM-index assumes constant alphabet; it uses exponential space in the alphabet size, σ. In this paper we show how the same ideas can be used to obtain an index needing O(Hkn) bits of space, with the constant factor depending only logarithmically on σ. Our space complexity becomes better as soon as σ log σ > log n, which means in practice for all but very small alphabets, even with huge texts. We retain the same search complexity of the FM-index. FM-index The FM-index [3] is based on the Burrows-Wheeler transform (BWT) [1], which produces a permutation of the original text, denoted by T bwt = bwt(T ). String T bwt is a result of the following forward transformation: (1) Append to the end of T a special end marker $, which is lexicographically smaller than any other character; (2) form a conceptual matrix M whose rows are the cyclic shifts of the string T$, sorted in lexicographic order; (3) construct the transformed text L by taking the last column of M. The first column is denoted by F . The suffix array A of text T$ is essentially the matrix M: A[i] = j iff the ith row of M contains string tjtj+1 · · · tn$t1 · · · tj−1. Given the suffix array, the search for the occurrences of the pattern P = p1p2 · · · pm is trivial. The occurrences form an interval [sp, ep] in A such that suffixes tA[i]tA[i]+1 · · · tn, sp ≤ i ≤ ep, contain the pattern as a prefix. This interval can be searched for using two binary searches in time O(m log n) [5]. The suffix array of text T is represented implicitly by T bwt. The novel idea of the FMindex is to store T bwt in compressed form, and to simulate a backward search in the suffix array as follows: Algorithm FM Search(P [1,m],T [1, n]) (1) c = P [m]; i = m; (2) sp = CT [c] + 1; ep = CT [c + 1]; (3) while (sp ≤ ep) and (i ≥ 2) do (4) c = P [i − 1]; (5) sp = CT [c] +Occ(T bwt , c, sp − 1)+1; (6) ep = CT [c] +Occ(T bwt , c, ep); (7) i = i − 1; (8) if (ep < sp) then return “not found” else return “found (ep − sp + 1) occs”.